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An automated system for organizing, present- 
ing, and manipulating medical images includes a 
database in which the medical images are structured 
into groups, each group including one or more im- 
age series, each image series including an ordered 
sequence of images which illustrate incrementally 
registered aspects of an anatomical target Image 
series are pre seated in their sequential order either 
in a monitor presentation format which displays each 
sequence in its entirety in a single monitor display 
container or which presents two or more image se- 
ries, image- by- image, in adjacent presentation areas 
of a series display container. The system includes 
a plurality of monitors in which all monitors, save 
one, produce display containers for image series pre- 
sentation. One monitor is reserved for displaying a 
working palette to which images of the image series 
displayed on the other monitors may be moved. The 
system activates a monitor in a plurality of monitors 
in response to movement of a cursor between moni- 
tors. An active monitor is indicated by presentation 
of a control panel. The system also provides heads- 
up presentation of control panel icons at a cursor 
location outside of the control panel by sequentially 
changing the shape of the cursor to the icon shapes 
for user selection. 
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DESCRIPTION 
AN AUTOMATED SYSTEM AND A METHOD 
FOR 

ORGANIZING. PRESENTING. AND MANIPULATING 
5 MEDICAL IMAGES 

TECHNICAL FIELD 

The invention relates to the organization, presentation, and manipulation of 
images. More particularly, the invention provides efficient, user-friendly means 
and procedures for presenting images of anatomical structure and the like for 

10 examination. 

BACKGROUND ART 

The invention is also concerned with the activation of a display container 
for medical images in a context of a plurality of display containers, and with the 
presentation of icons without distracting the attention of a user from medical 

15 images which are being examined. 

Magnetic resonance imaging (MRI) is an important, non-invasive imaging 
modality that is widely used by radiologists to examine internal anatomy to aid in 
the analysis of trauma and the diagnosis of disease. An MRI study provides a 
multi-planar representation of an anatomical target in the form of one or more 

20 image series. The images of a series may be parallel planar "slices" of the 
anatomical target which are incrementally registered along an imaging axis. 
Another imaging technology may produce image sequences which include non- 
parallel views that are incrementally registered about an axis of rotation or over 
a non-planar surface. 

25 MRI technology is well understood. See, for example, MRI OF THE 

KNEE by Jerrol H. Mink, et al, Raven Press Ltd. (New York, NY 1987) and the 
work by F.W. Wehrli, et al, entitled PARAMETERS DETERMINING THE 
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OCCURRENCE OF NMR IMAGES published by the General Electric Company, 
Medical Systems Operations, in 1983. 

In the prior art, the transparencies produced by an MRI system would be 
manually mounted in their series sequence on long light boxes where they would 

5 be read and annotated by radiologists. Following examination, the images would 
be physically stored in a patient's medical history folder. Recently, automated 
systems for archiving, retrieving, and presenting MRI images have been developed. 
In these systems, the images are conventionally converted to multi-bit, pixelated 
data representations which are formatted, stored, and retrieved using file 

1 0 management techniques. However, most conventional file management techniques 
are adapted for storage, retrieval, and presentation of documents, rather than 
images. Of course, these systems give even less consideration to the specialized 
requirements for storage and presentation of images showing internal anatomy. 
Even the addition of a directly-manipulated user interface in conventional 

15 image storage and presentation systems does little to adapt these systems to the 
special needs of radiologists who must consider and manipulate many images in 
particular ways for special purposes. Furthermore, each radiologist has a highly 
personal mode- of examination. For example, one radiologist may wish to examine 
a first sequence of transparencies in its entirety and then a second, related sequence 

20 before trying to correlate between individual images of the sequences. Another 
radiologist may wish to examine sequences in parallel by simultaneously 
considering images taken of the same anatomical plane under different conditions 
of exposure. Even currently-available database systems which have been adapted 
for storage and presentation of radiological images have not automated the 

25 presentation modalities of image series. Instead, a radiologist must provide for the 
individual retrieval and presentation of each and every stored image. In these 
systems, the images are individually identified and processed for storage and 
presentation without correlation to other images in their respective sequences. 



Patent provided by Sughrue Mion, PLLC - http://vMw.sughrue.com 



WO 94/16399 PCT/US93/12580 



3 



Furthermore, the existing systems do not provide for concurrent presentation of 
related image series. 

Moreover, the currently available automated image storage systems are 
awkward and difficult to use, providing little in the way of means for direct 
5 manipulation of image presentation formats and images which are displayed for 
analysis. 

SUMMARY OF THE INVENTION 

Therefore, it is a primary objective of this invention to provide an 
automated system for storage, retrieval, and presentation of medical images which 
10 is especially adapted for the presentation of medical image sequences and which 
affords the user with a flexible and responsive set of functions that permit direct 
manipulation of the modes of image presentation and of the presented images 
themselves. 

The invention is embodied in a computer display system which presents 
15 images of anatomical structure and the like for examination. The system includes 
the following combination: 

a first display container including a first preselected number of 
substantially rectangular presentation areas in a substantially rectangular 
array; 

20 a second display container including a second preselected number 

of substantially rectangular presentation areas in a substantially rectangular 
array; 

an image database including a plurality of images of anatomical 
structure, the images being separated into image groups in which: 
25 each image group is indexed by a unique group 

identification; and 

each image group is partitioned into one or more ordered 
image series, each ordered image series including a succession of 
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images which illustrate incrementally registered aspects of an 
anatomical target, each image series being ordered by assignment 
to each image in the image series of a position in a respective 
monotonically changing sequence; 
5 a mechanism connected to the image database and to the first and 

second display container and responsive to a group identification for 
displaying at least two image series of an image group indexed by the 
patient identification, wherein: 

each image series is displayed in the order of its respective 
10 sequence in a respective display container such that each 

presentation area of the respective display container includes no 
more than one image; or, 

all of the image series are displayed in one display container 
and each image series is displayed one image at a time in the order 
15 of its respective sequence in a respective presentation area of the 

display container. 

For direct manipulation of an interactive computer system with an output 
means for presenting visual displays wherein a user is presented with a visual 
display that includes a control display container (control panel), the invention 
20 further includes the following combination: 

a first bounded display area presented by the output means, the first 
display area including: 

a control panel; and 

a plurality of icons displayed in the control panel; 
25 a second bounded display area presented separately from the first 

display area by the output means; 

a cursor mechanism connected to the output means for displaying 
and moving a cursor in a display area; and 
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a functional mechanism coupled to the first and second display 
areas and the to the cursor mechanism for presenting the control panel in 
the second display area in response to movement of the cursor from the 
first to the second display area. 
5 Also, in an interactive computer system with an output means for 

presenting visual displays wherein a user is presented with a display that includes 
a directly-manipulated display container for presenting program output, the 
invention is embodied in a combination, in which: 

a cursor mechanism is connected to the output means for displaying 
10 and moving a cursor; 

a control display area is provided in the display in which a plurality 
of icons are displayed in a predetermined order; 

a pointing mechanism is provided, with connection to the cursor 
mechanism, for positioning the cursor and selecting icons in response to 
15 direct manual manipulation by a user; and 

an icon rotation mechanism is provided, with connection to the 
pointing mechanism and to the cursor mechanism, which responds to a 
position of the cursor inside the display container by changing the shape 
of the cursor at the position to the shape of an icon of the plurality of icons 
20 and by changing the shape of the cursor to the shapes of the other icons in 

the plurality of icons, such that the shape of the cursor is changed one 
shape at a time and in the predetermined order, in response to direct 
manual manipulation of the pointing mechanism. 

BRIEF DESCRIPTION OF DRAWING 
25 The invention, and its achievement of the above-stated objective, will be 

understood when the following detailed description is read in connection with the 
below-described drawings, in which: 
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Figure 1 is a block diagram illustrating basic components of an 
automated system for storing, retrieving, and displaying medical images; 

Figure 2 is a block diagram illustrating in greater detail certain 
components of the system of Figure 1 ; 
5 Figure 3 is an illustration of a display container output on a monitor 

of the system of Figure 1 for a presentation of medical images according 
to one mode of the invention; 

Figure 4 is an illustration showing a display container for 
presentation of medical images according to a second mode of the 
10 invention; 

Figure 5 is an illustration showing a working palette display 
container presented on a monitor of the system of Figure 1 ; 

Figure 6 illustrates the organization of a medical image database 
according to the invention; 
15 Figure 7 is a flow diagram illustrating how the database of Figure 

6 is accessed; 

Figure 8 is a flow diagram illustrating how the system of Figures 
1 and 2 presents images according to the first mode of the invention; 

Figure 9 is a block diagram illustrating presentation of images 
20 according to the first mode of the invention; 

Figure 10 is a flow diagram illustrating how the system of Figures 
1 and 2 presents images according to the second mode of the invention; 

Figure 1 1 is a block diagram illustrating presentation of images in 
a database of Figure 6 according to the second mode of the invention; 
25 Figure 12 is a flow diagram illustrating a procedure for indicating 

an active monitor according to the invention; 

Figure 13 is a flow diagram illustrating a procedure for rotating 
image manipulation icons according to the invention; 
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Figures 14a through 14d are illustrations of display containers 
showing icon rotation according to the procedure of Figure 13; 

Figure 15 is a flow diagram illustrating annotation rotation 
according to the invention; 
5 Figure 16 is an illustration showing a display container with 

annotation rotation according to the procedure of Figure 15. 

DISCLOSURE OF INVENTION 

The invention is intended to operate in connection with an image database 
which stores medical images for retrieval and presentation. Although the 

10 following description assumes that the images are data representations of pixelated 
images conventionally produced from the output of medical imaging technology 
such as MRI, X-ray, and CAT, this is not intended to limit such images to such 
sources. The inventors contemplate that the means and processes presented in this 
description would apply as well to a database of synthetically-created medical 

1 5 images. 

In the preferred embodiment and best mode of this invention, the medical 
images are, in fact, those produced from an MRI apparatus. As is known, such an 
apparatus produces series, or sequences of images, which represent successively 
deeper planar slices of an anatomical target. In the examples given below, the 

20 anatomical target is the head of a human patient and it is assumed that three 
distinct series of images have been taken of the target. Two of these series are 
termed "axial". The images of an axial series represent parallel, planar images of 
cross-sectional anatomy taken along an imaginary vertical axis of the head. Each 
axial series is generated to emphasize particular anatomical features by varying 

25 parametric values of the MRI process. In this regard, see Chapter 1 of the Mink 
et al reference. The two axial series described in this embodiment are the well- 
known axial Tl and axial T2 sequences. The third series is a sagittal sequence in 
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which the images are anatomical cross-sections lying in planes parallel to the 
median plane of the head. 

It is assumed, without further explanation, that axial and sagittal images 
obtained from an MRI apparatus will have been digitized and rendered into 
5 conventional scanned, pixelated data representations that can be processed by the 
computer system illustrated in Figure 1. In Figure 1, the system includes a DOS- 
based personal computer or work station 10 which has the capability to 
concurrently execute an application process 12 and a plurality of interface 
processes 14. The application process which, preferably, is written in the well- 

1 0 known C language, embodies processes and functions of the invention as described 
later in further detail. The interface processes 14 include commercially available 
programs as well as processes which can be constructed by the reasonably skilled 
computer programmer with the following description in hand. The interface 
processes 14 are connected to standard I/O devices such as a conventional trackball 

1 5 mechanism 1 6, a conventional QWERTY keyboard 22, and output devices 23 such 
as a full grey scale printer, a facsimile machine, and a modem (none of which is 
illustrated). The trackball mechanism 16 is a conventional picking device that 
includes a left button 18, a right button 19, and a rotation ball 20. The trackball 
mechanism 16 provides a user with the means for directly manipulating the 

20 position and functions of a cursor. The keyboard 22 is manually operated to input 
alphanumeric characters. The output devices 23 operate conventionally to provide 
tangible, visual results produced by the invention as described in further detail 
below. An image database 24 includes one or more conventional on-line storage 
devices for storage of medical image data representations which have been 

25 generated as described above. Therefore, the image database 24 includes not only 
the storage hardware, but also the stored images and all necessary indices. An 
output device 25 in the form of a multi-screen apparatus provides the means by 
which the images stored in the image database 24 are arranged and presented to 
a practitioner for examination, manipulation, and disposition. In the preferred 
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embodiment, the device includes three separate output monitors 26, 27, and 28 
which are connected by a coupling mechanism 29 to the interface processes 14 for 
control by the application process 12. 

Figure 2 shows certain architectural aspects of the best mode for practicing 
5 the invention in a system such as that illustrated in Figure 1. In Figure 2, the 
interface processes 14 include a conventional directly-manipulated user interface 
14a which is directly connected to a screen driver 14b. The screen driver 14b is 
connected to the trackball mechanism 16 and the keyboard 22, passing inputs from 
these devices by conventional means directly to the user interface 14a. The screen 
10 driver 14b also operates in conjunction with the user interface 14a to provide the 
pixel information necessary to draw and manipulate graphics on the monitors 26, 
27, and 28. 

The graphics include a cursor, display containers which give a visible 
structure for image presentation, and a control panel in which icons and function 

15 buttons are displayed. It is assumed that the user interface includes the ability not 
only to generate a cursor, but also to move the cursor among the monitors in 
response to trackball motion. It is further presumed that the user interface includes 
the ability to highlight icons and control buttons which have been "pointed to" by 
the cursor and "clicked" by a trackball mechanism button. The interface processes 

20 14 further include an image subsystem 14c and a low level video controller 14d. 
The image subsystem 14c accesses the image database 23 for retrieval of stored 
images. Image selection is made by the application process 12 and indicated by 
control signals coupled to the image subsystem 14c. In response to control signals 
which designate images to be retrieved, the image subsystem 14c obtains the data 

25 representations of the specified images and enters them into a high-capacity 
working memory 33 as directed by the application process 12. The application 
process 12 further specifies how and which images in the working memory 33 are 
to be output on the monitors 26, 27, and 28. Thus, the pixel information 
representing database images flows from the working memory 33, through the 
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image subsystem 14c to the monitors 26, 27, and 28. Control over the display 
apparatus 25 is provided by the coupling mechanism 29 in the form of a video 
card which has separately accessed and separately controlled memories for Red, 
Green, and Blue pixels. These memories are indicated by reference numerals 29a, 
5 29b, and 29c, respectively. The memories of the video card are controlled by a 
low-level video controller 14d, which receives control signals from the image 
subsystem 14c. The image subsystem 14c generates control input for the low-level 
controller 14d in response to control signals generated by the application process 
12. In the architecture illustrated in Figure 2, each of the memories 29a, 29b, 

10 and 29c is connected by a respective gun of a tri-color output to a particular video 
monitor. In particular, the Red memory 29a is connected by way of a Red (R) 
video output connector to the monitor 26, the Green memory 29b by a Green (G) 
video output connector to the monitor 27, and the Blue memory 29c by a Blue (B) 
video output connector to the monitor 28. The monitors 26, 27, and 28 are 

15 operated in the monochromatic mode and the memories 29a, 29b, and 29c are 
operated effectively as frame buffers for the monitors to which they are connected. 
This architecture merely reflects the design choices of the inventors in 
implementing the invention, and is not intended to limit the use of equivalent 
architectures. For example, instead of separate video monitors 26, 27, and 28, a 

20 designer might select an architecture which provides separate display containers 
(windows) on a single, large video monitor so that each window operates as a 
separate, independently-controlled monitor. 

The application program 12 employs object-oriented programming 
techniques to provide programming controls and cues to the user interface 14a and 

25 the imaging subsystem 14c. Controls and cues are in an object list 34 which is 
initiated and updated as needed by the application process 12. The object list has 
a conventional structure in that it includes a plurality of objects, each of which 
represents some feature or attribute of a display which is presented on one of the 
monitors 26, 27, or 28. Objects in the object list 34 are conventionally indexed. 
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Each object includes one or more application process functions and any. data 
necessary to establish machine state when the object is encountered or otherwise 
identified during system operation. 

The application process 12 maintains and increments an icon index 35 
5 whose output is provided to the user interface 14a for accessing the object list 34 
in a procedure described later in greater detail. 

IMAGE PRESENTATION 

The invention is practiced on a system architecture corresponding to that 
illustrated and discussed above. The invention is concerned with the presentation 

10 of one or more medical image series for consideration, analysis, and disposal by 
a user such as a radiologist. The invention includes at least two modes for 
presenting medical image series. The first mode is illustrated in Figure 3. 

Figure 3 illustrates what is termed the "monitor" mode of image 
presentation. Figure 3 illustrates a rectangular display container which is produced 

15 on one or more monitors for presentation of image series. The series shown in 
Figure 3 is an axial T2 series including 18 images. The display container is 
subdivided into a rectangular array of 20 presentation areas. In the monitor mode 
of image presentation, an image series is shown in a single display container in its 
sequence order such that each presentation area of the display container includes 

20 one image of the sequence. Only one image series at a time is shown in any 
display container. Separate series are shown in separate display containers. The 
axial T2 sequence illustrated in Figure 3 is ordered by assignment to each image 
in the series of a monotonically increasing number. The number assigned to an 
image represents the image's position in the sequence of which it is a member. 

25 In Figure 3, therefore, the image in the upper left-hand presentation area is the first 
image of the axial T2 sequence, with the next image in the sequence being 
assigned the number 2 and being displayed in the next presentation area of the 
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display container to the right of the presentation area where image number 1 is 
shown. 

In the monitor mode of the presentation, a display container such 
as that illustrated in Figure 3 is output on all but one of the monitors or monitor 
5 equivalents supported by the system. Thus, in Figures 1 and 2, a display container 
such as shown in Figure 3 would be presented on, for example, monitors 26 and 
27, with monitor 28 being reserved for presentation of a work product palette as 
described below. This permits more than one image series to be presented, with 
each image series being presented in its sequence order in a presentation area of 

10 a respective display container on a particular monitor. Since co-relative series 
such as axial Tl and axial T2 typically consist of an identical number of images 
taken at identical planes, identical side-by-side presentations of axial Tl and T2 
series on adjacent monitors in the manner illustrated in Figure 3 contributes 
significantly to a radiologist's ability to differentially analyze the images. 

15 Figure 4 illustrates a second presentation mode of the invention which is 

termed the "series" mode. In the series mode, a display container comprising a 
rectangular array of rectangular presentation areas is displayed on one or more 
monitors. For example, the display container in Figure 4 includes four 
substantially rectangular presentation areas numbered 1, 2, 3, and 4 in the drawing. 

20 In the invention, each presentation area of a series mode display container is 
employed to display an image series, one image at a time, in the order of its 
respective sequence. In this regard, consider presentation area 2 in which an axial 
T2 series is illustrated. Assume that the axial T2 series is identical to that 
illustrated in Figure 3. Thus, the images of the axial T2 series would be shown 

25 in presentation area 2 beginning with image 1 and continuing sequentially until 
image 18 is shown. Assume next that a co-relative axial Tl series comprising 18 
images is presented in presentation area 1 in the display container of Figure 4. An 
important feature of the invention termed "coupling" is employed to synchronize 
the presentation of the axial Tl and T2 series so that whenever an image of one 
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series is changed to the next image in the series, the other series is changed 
identically to display the image of the other series occupying the same sequence 
position. Thus, if presentation area 1 in the display container of Figure 4 displays 
image 1 of the axial Tl series and then riffles through the images of the series, the 
5 display in presentation area 2 would identically riffle through the images of the 
axial T2 series, beginning with image 1. The invention contemplates that the 
coupling feature of the series mode of presentation will synchronize image 
presentation only between co-relative image series. Thus, for example, an axial 
Tl series would be coupled for a presentation with an axial T2 series, but not with 

10 a sagittal Tl series such as that shown in presentation area 4 of the display 
container in Figure 4. 

The series mode of presentation is practiced by presenting the display 
container of Figure 4 on all available monitors or monitor equivalents except one, 
which is also reserved for the working palette display container. 

15 Refer now to Figure 5 for an illustration of the working palette display 

container. This display container is presented on one monitor of a multi-monitor 
system, such as monitor 28 in Figures 1 and 2. The working palette display 
container has a substantially rectangular aspect and is subdivided into a rectangular 
array of individual presentation areas. The working palette is provided to receive 

20 individual images which are picked and copied from presentation areas in monitor 
mode or series mode display containers. The working palette allows a radiologist 
to assemble the images which are deemed to be the most important in illustrating 
trauma or disease for provision to a referring physician. For example, the axial 
image displayed in presentation area 3 of the working palette display container 

25 may have been copied there from presentation area 12 in the monitor mode display 
container of Figure 3. 

Figure 6 illustrates the structure of the image database which enables the 
efficient storage and retrieval of image series. When an MRI apparatus is used to 
generate images, examination protocols normally provide for identification of the 
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patient, and set out the number and types of sequences which are to be taken and 
the anatomical target of interest. From this information, a unique composite 
patient identifier (ROOTNAME) is algorithmically derived. A patient, thus, has 
a unique ROOTNAME for each MRI examination. A patient identification, the 
5 corresponding ROOTNAME, and an identification of a referring physician are 
entered for each patient examination into a database file 50 entitled 
TATIENT.dB". For any patient identification entered in the file 50, its 
corresponding ROOTNAME points to an examination file such as the file 53, 
which is entitled "ROOTNAME. PAT". Each examination file 53 identifies an 

10 image group including one or more image series obtained during an examination. 
Each image series in an image group is indexed to a set of sequentially named or 
numbered files in which the images of the series are contained. For example, in 
the ROOTNAME.PAT file 53 illustrated in Figure 6, the image group includes 
axial Tl, axial T2 ; and sagittal Tl series. The axial Tl series consists of n 

1 5 consecutively-numbered files, each containing a respective image of the series such 
that the sequence position of the named file corresponds to the sequence position 
of the image it contains. The image files are named ROOTNAME.XXX. Thus, 
image 1 in the axial Tl series has a file name ROOTNAME. 1, the second image 
in this series is in a file named ROOTNAME.2, and so on. Similarly, the n 

20 images of the axial T2 series are stored in files consecutively numbered as 
ROOTNAME.m through ROOTNAME.(m+n). When a patient is identified to the 
application process 12, the application process determines a ROOTNAME value 
for that patient, obtains a ROOTNAME.PAT file, and commands the image 
subsystem to extract the image series contained in the ROOTNAME file sequence 

25 defined by the ROOTNAME.PAT file and place them in linked list form in the 
working memory 33. The application process 12 then presents the image series 
to a radiologist and provides the radiologist's work product assembled in a 
working palette, to the referring physician. 
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The file-naming rule for image series is important to the invention because 
it preserves the unique identity, and the order, of each image series. For any given 
ROOTNAME, an image series can be identified by reference to the ROOTNAME 
file 53 and manipulated, as a unit and independently of any other image series, by 
5 the system of Figures 1 and 2. This provides a very significant advantage over the 
existing systems which treat each image as an independent unit unconnected with 
any other image in its sequence or with any other image in any other series or 
group. 

As Figure 6 illustrates, the PATIENT.dB file includes a REFERRING 

10 PHYSICIAN column containing values which index to entries in a REF.DOCS file 
52. The REF.DOCS file 52 lists referring physicians. Each referring physician 
entry includes a set of functions specified by the identified referring physician 
which are to be executed when a radiologist indicates that examination of an image 
group is completed by pressing a DONE button, which is explained later in more 

15 detail. A RADSTAFF.dB file 63 includes at least seven columns. In the first 
column (RAD NAME) are listed the names of radiologists having access to the 
system. The second through sixth columns comprise a format specification for the 
monitors in which the image series are to be displayed. The seventh column 
includes a working palette specification. The format specification, columns two 

20 through six of the Table 63, include a MODE column, identifying the monitor or 
series mode of presentation, a MATRIX column setting forth the number of rows 
and columns in the display containers, and a COUPLE column specifying whether 
the coupling function is invoked for synchronization of two or more image series 
in a series mode of presentation. The format specification further includes 

25 SHRINK and EXPAND columns which specify whether the rows and columns of 
the matrix are to be adapted to predefined dimensional parameters for display 
containers. The seventh column specifies the matrix size of the working palette. 
The application process also has access to a set of annotation lists 60 and to the 
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set of image functions that are linked to objects which are in or which may be 
added to, the object list 34 of Figure 2. 

Figure 7 illustrates the procedure executed by the application process 12 to 
initialize the system of Figures 1 and 2 for presentation of images. The procedure 
5 starts at step 70 by receipt of an indication that the PATIENT.dB file is to be read. 
In step 71, a list illustrating the PATIENT.dB file is constructed and presented on 
one of the monitors. It is assumed that the user can select the name of a patient 
simply by moving a cursor to the patient's name in the displayed list and clicking 
on the name by depressing and releasing a trackball button. The decision 73 
10 awaits selection of a patient. In decision 73, the negative exit can be manually 
selected. If the negative exit is taken from decision 73, the procedure is exited. 
If a patient is selected, the positive exit is taken from decision 73 and a search is 
conducted of the database for a ROOTNAME.PAT file for this patient in step 74. 
If the file does not exist, a warning message to the operator is displayed on the 
15 monitor in step 75. Otherwise, the ROOTNAME.PAT file is provided to the 
application process which extracts the image group organization from the file in 
step 77, and passes the appropriate control signals to the imaging system in step 
78 to enable the imaging subsystem to copy the images named in the 
ROOTNAME.PAT file from the image database to the main memory. In step 79, 
20 a list illustrating the RADSTAFF.dB file is presented in a manner permitting a 
radiologist to log on by pointing to and clicking his name in the list. When the 
radiologist's name is identified in this manner, the formal specification and 
working palette matrix size in the same RADSTAFF.dB row are provided to the 
application process 12 which enables it to initialize control blocks for each 
25 monitor, as well as the object list, in step 80. The object list includes a full 
specification of the graphics objects required in the first monitor for the mode and 
the matrix size, and palette size specified in the RADSTAFF.dB entry selected by 
the user. The first monitor is then initialized with graphics and images in steps 81 
and 82. Once the first monitor has been initialized, the object list is updated for 
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specifying the graphics of the second monitor, and then is updated once again to 
specify the working palette graphics for the third monitor. The third monitor is 
initialized with graphics in step 83. The application process then executes the 
appropriate presentation mode procedure. In the preferred embodiment, the 
5 monitors 26 and 27 in Figures 1 and 2 are formatted with display containers of the 
specified mode and matrix size and populated with images and then the monitor 
28 is formatted with the specified working palette. 

Referring once again to Figures 1 and 2, when the object list 34 is 
initialized or updated, the application process provides a "SELECT MONITOR" 

10 control signal to the image subsystem 14c. This signal is appropriately 
conditioned to indicate the monitor to which control is to be transferred. In 
response, the image subsystem configures the low level controller 14d to enable 
the appropriate one of the memories 29a, 29b, or 29c. For example, when the 
object list is initialized, the SELECT MONITOR control signal forces the low 

15 level controller 14d to enable the Red memory 29a to receive the appropriate 
display container graphics established in the object list 34. The user interface 14a 
inspects the object list 34 to determine the graphics objects for which graphics 
pixels must be generated and provided by the screen driver 14b. The screen driver 
14b outputs the specified graphics pixels in parallel to all of the memories 29a, 

20 29b, and 29c. The pixels, however, are read into only the currently-enabled 
memory. 

Once the series of an image group have been entered in sequence order into 
the working memory 33 and the display containers have been initialized, the 
application process produces the appropriate image system control signals to 
25 present an initial image display in the specified presentation mode. If, for 
example, the monitor mode has been specified, graphics pixels for presenting a 
display container such as that illustrated in Figure 3 in the specified matrix size are 
entered into the Red memory 29a of the video card together with image pixels for 
the first series of the image group. When the memory 29a is appropriated loaded 
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with graphics and image pixels, the application process issues a SELECT 
MONITOR control signal to the low level controller 14d through the image 
subsystem 14c, which disables data entry to the Red and Blue memories 29a and 
29c and enables data entry into the Green memory 29b. Graphics pixels and 
5 image pixels for the second series of the image group are entered into the memory 
29b and the SELECT MONITOR signal is once again issued so that graphics 
pixels for the working palette can be entered into the Blue memory 29c. Next, the 
object list is updated to describe the Red memory 29a, the Red memory 29a is 
enabled, and the cursor is generated. The cursor pixels are fed to the enabled 
1 0 memory and updated therein when the cursor is moved. 

At this point, display containers with images would be presented on the 
monitor 26 and on the monitor 27, while an empty working palette would be 
displayed on the monitor 28. 

Returning to Figure 3, assume that the monitor display container and image 
1 5 series in this figure are displayed on monitor 26 as shown in the figure. Thus, the 
presented display includes a rectangular area adjacent the right edge of the display 
container. This area is referred to as the "control panel" and contains a plurality 
of function buttons (DONE, CHANGE SERIES, and PRINT), a set of image 
manipulation icons immediately below the function buttons, and a set of annotation 
20 icons below the image manipulation icons. Any of the function buttons and any 
of the image manipulation or annotation icons can be activated by "point and 
click" using the trackball to point with the cursor, and the left cursor button to 
click on a button or icon. 

The function buttons represent application process functions that can be 
25 activated by pointing and clicking. The DONE button, when activated, invokes 
the "DONE" functions of the referring physician. The CHANGE SERIES button, 
when activated, deletes the series presently displayed in the display container and 
enters the images of the next series listed in the ROOTNAME.PAT file. The 
PRINT button invokes an application process function which prints a copy of the 
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screen. As also illustrated in clashed outline in Figure 3, another button may be 
presented in a control panel between the DONE and CHANGE SERIES buttons. 
The button may be a NEXT or a PREVIOUS button. When the image series 
includes more images than there are presentation areas in the display container, the 
5 NEXT button replaces the currently-displayed subset of consecutively numbered 
images of the series with the following subset. So long there are images 
remaining to be presented, the NEXT button will be presented. When the last 
subset of the image series is displayed in the display container, the PREVIOUS 
button is displayed. When activated, the PREVIOUS button returns the previous 

10 subset of series images in the display container. It should be evident that the 
NEXT and PREVIOUS buttons are displayed only in monitor mode control panels. 

The set of image manipulation function icons is positioned immediately 
beneath the PRINT button. These icons control various image manipulation 
functions. The upper left-most icon is a magnifying glass and controls a 

1 5 conventional zoom function which magnifies or minifies an image in response to 
trackball motion. Immediately beneath the zoom icon is a riffle icon in the form 
of a spaced set of pages. In response to trackball motion, the riffle function 
shuffles through an image series in forward or reverse sequence. Beneath the riffle 
icon is an icon representing illumination which adjusts the contrast or brightness 

20 of an image in response to trackball motion. Immediately to the right of the zoom 
icon is a roam icon consisting of four outwardly-directed arrows. The roam 
function is used in conjunction with the zoom function and moves a magnified 
image under a viewing aperture in response to trackball motion. The icon in the 
shape of a movie camera immediately beneath the roam icon denotes the cine 

25 function, which is an automated riffle that, once selected, automatically steps 
through an image series in sequence order. The speed of the cine function is 
controlled by the trackball. 
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A set of annotation icons consisting of four arrows and A and X and a 

rectangular box is presented immediately beneath the image manipulation icons. 

These icons are discussed later in more detail. 

As Figures 3 and 4 illustrate, the cursor has a default shape in the form of 
5 a hollow arrow. 

Refer now to Figures 3, 8, and 9 for an understanding of a monitor 
presentation format procedure according to the invention. Once the displays in the 
monitors 26, 27, and 28 have been initialized for monitor presentation as described 
above, the monitor format procedure begins. This procedure executes in response 
1 0 to cursor location and trackball button activation. Thus, in step 90, beginning with 
monitor 26, for example, and using the cursor location and sensing button 
activation, the procedure determines whether one of the image manipulation icons 
has been selected in decision 91. If so, the image function is performed at the 
cursor location. Otherwise, the negative exit is taken from decision 91 and the 
15 status of the CHANGE SERIES button is inspected in decision 92. If the button 
function has been activated, the next series listed in the ROOTNAME.PAT file is 
moved into the display container grid of the monitor where the cursor is located 
when the left button of the trackball mechanism is clicked. Now, a series counter 
(not shown) for the monitor is updated to identify the series currently displayed 
20 on the monitor. If the function has not been activated, the negative exit is taken 
from decision 92. In decision 93, the application process determines whether a 
working palette grid location has been selected for placement of an image. In this 
regard, the application process maintains a working palette grid pointer which is 
updated to the next working palette grid location when an image is moved to the 
25 current one. The grid pointer can also be forced to a grid location by moving the 
cursor from the monitor where it is currently located to the monitor where the 
working palette is displayed and pointing and clicking a grid location in the 
working palette display container. Preferably, a working palette grid location is 
selected by clicking the left button of the trackball mechanism. (Once an image 
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has been moved to a preselected working palette grid location, the grid pointer is 
reset to indicate the next empty grid location which was available before user 
selection.) The cursor is then returned to a monitor which shows an image series. 
Decision 94 assumes that the cursor is in a presentation area of the monitor mode 
5 display container. If the right button of the trackball mechanism is clicked, the 
application process infers that the user wants the image in the presentation area 
moved to the working palette grid location indicated by the palette grid pointer. 
In this case, the image at the cursor location and any annotation graphics which 
it contains are copied into the working palette grid at the location indicated by the 
10 working palette grid pointer. Refer to Figure 9 for an understanding of how the 
application process accomplishes this step. 

In Figure 9, three monitor control blocks 100, 102, and 103 are illustrated. 
The application process maintains the control blocks, each of which completely 
describes in object form the current display on a particular monitor. For example, 
15 the monitor control block 1 00 lists the identification and location of each graphical 
object displayed on the monitor 26. This would include a complete object 
description of a monitor mode display container, a control panel, if a monitor is 
active, and the images of the series presented in the display container. For 
example, consider the first image of an axial Tl series which is listed in grid 
20 location 1, corresponding to the upper left-hand presentation area in the display 
container of Figure 3. The monitor control block 100 also lists the graphics 
displayed at the grid location. When an image is to be moved to the working 
palette it is copied from the object indexed by the current cursor location in the 
control block of the monitor where the cursor is located to the palette monitor 
25 control block at the grid location indicated by the working palette grid pointer 104. 
In order to effect the update on the working palette display container presented on 
the palette monitor 28, the application process, using the palette monitor control 
block, updates the object list to describe the updated working palette display 
container and issues a SELECT MONITOR control signal. In response to the 



WO 94/16399 



PCT/US93/12580 



22 



SELECT MONITOR control signal, the image subsystem accesses the working 
memory to obtain the pixels for the image to be moved and enters them into the 
Blue memory 29c which has been enabled by the SELECT MONITOR signal. At 
the same time, the screen driver 14b provides any annotation pixels for the image 
5 which may be described in the object list. This causes the selected image and 
accompanying graphics to be added to the working palette presented on the 
monitor 28. When entry of the image and graphics pixels in the Blue memory 29c 
is complete, the application process updates the object list to describe the display 
presented in the monitor where the cursor is located and issues a SELECT 
10 MONITOR signal to enable the corresponding color memory. 

In decision 95, the application process inspects the cursor location to 
determine whether the cursor has been moved to another monitor. If so, the object 
list is updated to the configuration of the monitor where the cursor is located and 
the SELECT MONITOR control signal is conditioned to enable the corresponding 
15 color memory for the new monitor. 

In decision 96, if a DONE button is selected, the application program 
executes the DONE functions for the referring physician. These may include, for 
example, simply printing out a hard copy of the working palette display container 
and entering the printout into the patient's file, faxing a data or an image copy of 
20 the working palette display container to a predetermined location, or transmitting 
a data representation of the working palette monitor through a modem to a 
predetermined location. 

Refer now to Figures 4, 10, and 11 for an understanding of a series 
presentation format procedure according to the invention. Once the displays in the 
25 monitors 26, 27, and 28 have been initialized for series presentation as described 
above, the format procedure begins. This procedure executes in response to cursor 
location and trackball activation. Thus, in step 1 10, beginning with monitor 26, 
for example, using cursor location and sensing button activation, the procedure 
determines whether one of the image manipulation icons has been selected in 
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decision 111. If so, the image manipulation function is filtered at decision 1 12 to 
determine whether the riffling function has been invoked with the coupling feature 
selected. If not, the negative exit is taken from decision 112 and the image 
manipulation function is performed in the presentation area where the cursor is 
5 located. On the other hand, if the riffling function has been invoked with coupling 
activated, the positive exit is taken from decision 112. Here, the images of the 
series being output through the presentation are where the cursor is located are 
riffled (in response to trackball motion) in the presentation area where the cursor 
is located. In addition, the images of any coupled series are riffled at the 
10 presentation area where the coupled series is displayed. The riffling is in 
synchronism with riffling the series at the cursor location. Decision 113 tests the 
CHANGE SERIES button of the control panel. If the button has been activated, 
the procedure waits until the cursor is moved to a display container presentation 
area and the left button is clicked. Then, the series is changed to the next 
15 ROOTNAME.PAT file image series. When the series is changed, the first image 
of the next series is displayed at the presentation area where the cursor is located. 
Step 116 moves images to the working palette display container in the manner 
described above for the monitor presentation format. Next, monitor processing and 
DONE button processing are performed in decisions 117 and 118 as discussed 
20 above in connection with Figure 8. 

Figures 4 and 11 illustrate how the working palette is populated with 
images during the series presentation mode. In the series presentation mode, the 
application process maintains a monitor control block for each of the monitors 26, 
27, and 28. The monitor control blocks are, respectively, 120, 121, and 122. 
25 Assume that the series presentation display containers such as the display container 
illustrated in Figure 4 are presented on the monitors 26 and 27. For each monitor, 
the control block must list the principal objects which, for the display container of 
Figure 4, are the four presentation areas of the control panel and the icon control 
panel. The control block links a respective series with each of the presentation 
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areas in the IMAGES column of the block. In the COUPLING column, a related 
image series is listed if the coupling function has been invoked. The GRAPHICS 
column lists the graphics for, and in, each presentation area. The monitor control 
block 121 fully describes the display container and control panel displayed on the 
5 monitor 27, while the control block 122 fully describes the working palette 
presented on the monitor 28. A grid pointer 122a is used to denote the current 
working palette grid location in the manner described above in connection with 
Figure 9. 

ACTIVE MONITOR INDICATION 
10 The invention also includes the ability to indicate, by the control panel, 

which monitor is currently active. In this regard, the "active" monitor is most 
likely the one wherein the cursor is currently located. The configuration of the 
active monitor is fully described in the object list; user inputs are received through 
it; and application response is provided through it. Manifestly, it would be 

1 5 inefficient and distracting to the user to have to remember which monitor is active 
by devoting full attention to the cursor's location. Positioning the cursor in a 
particular monitor might, in some instances, not indicate the active monitor. In 
addition, as described later, the shape of the cursor may change, causing the user 
to lose track of it and the active monitor. 

20 In the invention, the active monitor is indicated by the presence of a control 

panel. Preferably, when a user moves a cursor to a monitor boundary which is 
shared with another monitor, the cursor is moved from the one monitor (which is 
called the "last" monitor) to the monitor which is adjacent the boundary of the last 
monitor (which is called the "next" monitor). The last monitor is then deactivated 

25 and the next monitor is activated. In response to movement of the cursor from the 
last monitor to the next, the control panel in the last monitor is turned off and the 
control panel is presented in the next monitor. 
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The procedure for performing the active monitor indication function is 
illustrated in the flow diagram of Figure 12. Initially, the cursor position is read 
in step 123. When the cursor location is the same as the location of a boundary 
object defined in the object list of the current monitor, the application process 
5 prepares to deactivate the present monitor. In this regard, the positive exit is taken 
from the decision 124 and the application process issues control signals which turn 
off the control panel of the last monitor in step 125, reconfigures the object list to 
define all of the objects, including the boundaries, in the next monitor in step 126, 
and then executes a series of next monitor routines in step 127. One next monitor 

10 routine is conditioning of the SWITCH MONITOR signal to enable the color 
memory for the next monitor. Additional next monitor routines are shown in step 
128. In step 128, cursor graphics pixels are configured to place the cursor adjacent 
the boundary in the next monitor which is closest to the last monitor. Next, the 
control panel is turned on in the next monitor and the procedure returns to step 

15 123. 

The control panel deactivation/activation sequence to indicate monitor 
activation is not necessarily limited to turning off the control panel in the last 
monitor and turning on the control panel in the next monitor. Other visual cues 
may be used. For example, the control panel colors may be inverted or changed 
20 in intensity. 

ICON ROTATION 

The invention also provides a "heads up" feature which permits a user to 
select an image manipulation icon without shifting attention from a presentation 
25 area to the control panel. This is accomplished by moving the cursor one time to 
an image manipulation icon in the control panel, changing the shape of the cursor 
to that icon, moving the cursor back to a presentation area, and rotating the shape 
of the cursor at that location through the succession of image manipulation icon 
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shapes in response to motion of the trackball, and selecting an icon function at the 
location where the cursor shape has been changed. 

Refer to Figures 13 and 14a-14d for an understanding of the icon rotation 
function of the invention. Initially, in decision 131, the cursor position read at 
5 step 130 is evaluated to determine whether it is over one of the image 
manipulation icons in the control panel. Positioning the cursor over one of these 
icons and clicking the left button causes the cursor to change its shape from a 
default shape (preferably, a hollow arrow) to the shape of the selected icon and 
results in illumination of the icon at the control panel as provided in step 133. 

10 The result of performing step 133 is illustrated in Figure 14a where the shape of 
the cursor has been changed to the riffle icon shape comprising a set of pages. 
Selection of the icon at its control panel location will not immediately invoke the 
corresponding function because the image manipulation functions are intended to 
execute with respect to a selected image. Thus, to invoke the selected function, 

15 the cursor, now in the shape of the selected icon, must be moved to one of the 
presentation areas. Therefore, in step 134, the position of the cursor (now in the 
shape of an icon) is read and when decision 135 determines that the cursor is in 
an image display container, the positive exit is taken and decision 136 checks the 
status of the left button on the trackball mechanism. If the left button is not down, 

20 the procedure loops through 134, 135, and 136 until it detects depression of the 
left button. When the left button is held down and the trackball is moved, the 
shape of the cursor is successively changed to the shapes of the image 
manipulation icons in their order of presentation in the control panel. This is 
illustrated in Figure 14b where the cursor, in the shape of the riffle icon, is moved 

25 to the lower left corner of the top right presentation area of the illustrated series 
display container. Assuming that the user continues to hold the left button down 
while moving the trackball (136, 137, 138), the icon counter (indicated by 
reference numeral 35 in Figure 2) is incremented or decremented in response to 
the trackball motion. The threshold check in step 139 indicates when the trackball 
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has moved far enough to increment to the next value of the icon counter. The 
icon counter indexes to the portion of the object list which contains the image 
manipulation icons. These objects are numbered consecutively in the object list 
so that, by constraining the icon counter to cycle through a count sequence which 
5 corresponds to the index numbers of the icon objects, the user interface and video 
driver are enabled to correspondingly cycle through the icon shapes at the cursor 
location. In step 140, the shape of the cursor is changed at its location to the next 
image manipulation icon shape in sequence. 

For example, consider Figures 14b-14d where the cursor initially has the 

10 shape of the riffle icon. Assume that the trackball is moved in a certain direction, 
resulting in change of the icon counter to a value which corresponds to the 
magnifying glass icon shape. The result is illustrated in Figure 14c, where the 
cursor shape is changed to the magnifying glass. Further movement of the 
trackball in the same direction results in change at the icon counter to a value 

15 corresponding to the image motion icon consisting of four outwardly directed 
arrows shown in Figure 14d. As illustrated in step 140, and in Figures 14b-14d, 
each time the cursor shape is changed to that of the next icon in the icon rotation 
procedure, the corresponding icon is illuminated on the control panel at the cursor 
location. The decision in step 141 continues the icon rotation loop until the left 

20 button is released. In this regard, for so long as the left button is held down and 
the negative exit is taken from decision 141, the decision at 142 either loops back 
to the decision 141 if there is no trackball motion or back up to step 138 if there 
is further trackball motion. If, after icon rotation, the left button is released, the 
positive exit is taken from decision 141 and a click of the left button is searched 

25 for by entering decision 136 through C. 

The sequence 136, 137, 145 signifies a left button click to select an icon 
function, with the function being performed in step 146. This sequence can be 
entered by selecting an icon shape at the control panel, moving the cursor in the 
form of the shape to a display container, and then clicking the left button, as at 
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136, 137, 145. Alternatively, the clicking sequence is detected out of the positive 
exit from decision 141 after icon rotation when the left button is released. Note 
that the clicking sequence is exited if trackball motion is detected in step 137 
following depression of the left button in step 136 before the left button is released 
5 (step 145). Any appropriate reset sequence will restore the cursor to its default 
shape. 

IMAGE ANNOTATION 

Image annotation is provided by the set of annotation icons which is 
beneath the image manipulation icon set in the control panel. Arrows may be 

1 0 dragged from this icon set and selectively dropped on images by moving the cursor 
to the arrow, clicking the left button to change the shape of the cursor to the 
arrow, and moving the cursor in the arrow shape to the desired location. At the 
desired location, clicking the left button will "drop" the arrow at the location. Any 
appropriate RESET routine will return the cursor to its default shape. 

1 5 An image can be annotated by selecting the A icon in the control panel, 

dragging the icon to a location where annotation is desired, clicking the left button 
of the trackball mechanism and then using the keyboard to annotate the image at 
the cursor location. 

Figures 15 and 16 illustrate a procedure executed by the application process 

20 for automatically annotating a figure using the annotation lists indicated by 
reference numeral 60 in Figure 6. Initially, it is asserted that the application 
process includes logic capable of selecting an annotation list which is appropriate 
to the anatomical target. Thus, for example, a sagittal image sequence of a portion 
of the spine would identify an annotation list which included shorthand annotations 

25 commonly employed to identify individual vertebra. The procedure of Figure 1 5 
provides for the rotation of an annotation list through the annotation box which is 
illuminated in the annotation icon group in Figure 16. When this icon is selected, 
the cursor assumes its shape and is moved in response to the trackball to the 
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desired location. At the desired location, the left button of the trackball is 
depressed in order to couple the annotation list to the box at the cursor location 
and in the control panel. For so long as the left button is depressed, motion of the 
trackball will rotate the listed annotations through the annotation box outlined in 
5 the image and in the control panel. When the appropriate annotation is rotated 
into the box, the left button is released. When the left button is clicked again, the 
annotation will be dropped in the shape of the annotation box at the cursor location 
and the cursor's shape will be changed back to the default shape with the tip of 
the cursor pointing to the lower left-hand corner of the annotation box in the 

10 image. This is illustrated in Figure 16 and the procedure laid out in Figure 15. 

In Figure 15, when the cursor is clicked at the annotation box, positive 
exits are taken from decisions 150 and 151, the box icon is illuminated at the 
control panel and the cursor in the shape of the box is moved to a desired location. 
When the left button is depressed while the cursor is in an image display container, 

15 the positive exits are taken from decisions 152 and 153 and the annotation list is 
rotated through the box shape in steps 154. When the left button is released and 
clicked again, the annotation is dropped at the last cursor position and the cursor 
shape is restored. 

The X icon in the annotation set (Figure 16) is used to delete annotations 
20 from an image. Initially, the X icon is clicked using the cursor. This illuminates 
the X icon and changes the cursor to the icon shape. The cursor is moved over 
the annotation to be deleted and the left trackball button is clicked. This removes 
the annotation from the object list and from the image, restores the cursor to its 
default shape, and turns off the X icon in the control panel. 
25 Clearly, other embodiments and modifications of this invention will occur 

readily to those of ordinary skill in the art in view of these teachings. For 
example, cursor movement and clicking can be controlled from a keyboard. 
Furthermore, the images which are stored in the database may be actual images 
taken from the human patient, may be images taken from a veterinary subject, or 
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may be synthesized images. Therefore, this invention is to be limited only, by the 
following claims, which include all such embodiments and modifications when 
viewed in conjunction with the above specification and accompanying drawings. 
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CLAIMS 

1. A computer display system for presenting images of anatomical 
structure and the like for examination, including: 

a first display container means for presenting a first preselected 
5 number of substantially rectangular presentation areas in a substantially 

rectangular array; 

a second display container means for presenting a second 
preselected number of substantially rectangular presentation areas in a 
substantially rectangular array; 
10 an image database including a plurality of images of anatomical 

structure, the images being separated into image groups, in which: 

each image group is indexed by a unique group 
identification; and 

each image group is partitioned into one or more ordered 
15 image series, each ordered image series including a succession of 

images which illustrate incrementally registered aspects of an 
anatomical target, each image series being ordered by assignment 
to each image in the image series of a position in a respective 
monotonically changing sequence; and 
20 means connected to the image database and to the first and second 

display container means and responsive to a group identification for 
displaying at least two image series of an image group indexed by the 
patient identification; 

each image series being displayed in the order of its 
25 respective sequence in a respective display container such that each 

presentation area of the respective display container includes no 
more than one image. 

2. A computer display system for presenting images of anatomical 
structure and the like for examination, including: 
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at least one display container means for presenting a preselected 
number of substantially rectangular presentation areas in a substantially 
rectangular array; 

an image database including a plurality of images of anatomical 
5 structure, the images being separated into image groups, in which: 

each image group is indexed by a group identification; and 
each image group is partitioned into one or more ordered 
image series, each ordered image series including a sequence of 
images which illustrate incrementally registered aspects of an 
10 anatomical target, each image series being ordered by assignment 

to each image in the image series of a position in a respective 
monotonically changing sequence; and 

means connected to the image database and to the display container 
means and responsive to a group identification for displaying at least two 
15 image series of an image group indexed by the patient identification; 

each image series being displayed one image at a time in the 
order of its respective sequence in a respective presentation area. 
3. In an interactive computer system with an output means for 
presenting visual displays wherein a user is presented with a visual display that 
20 includes a control display container (control panel), a combination, comprising: 
a first bounded display area presented by the output means, the first 
display area including: 

a control panel; and 

a plurality of icons displayed in the control panel 
25 a second bounded display area presented separately from the first 

display area by the output means; 

a cursor means connected to the output means for displaying and 
moving a cursor in a display area; and 
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means coupled to the first and second display areas and . to the 
cursor means for presenting the control panel in the second display area in 
response to the cursor moving from the first display area to the second 
display area. 

5 4. The combination of Claim 3, further including: 

a first boundary in the first display area; and 
a second boundary in the second display area; 
wherein the means is further for moving the control panel in 
response to the cursor moving across the first boundary toward the second 
10 boundary. 

5. The combination of Claim 4, wherein the means is further for 
positioning the cursor just inside the second boundary in response to the cursor 
moving across the first boundary toward the second boundary. 

6. The combination of Claim 3, wherein the means is further for 
15 activating the control panel in the second display area and deactivating the control 

panel in the first display area in response to the cursor moving from the first 
display area to the second display area. 

7. The combination of Claim 3, wherein the means is for displaying 
the control panel in the second display area and for turning off display of the 

20 control panel in the first display area. 

8. In an interactive computer system with an output means for 
presenting visual displays wherein a user is presented with a display that includes 
a directly-manipulated display container for presenting program output, a 
combination, comprising: 

25 a cursor means connected to the output means for displaying and 

moving a cursor; 

a control panel in the display in which a plurality of icons are 
displayed in a predetermined order 
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a pointing means connected to the cursor means for positioning the 
cursor and selecting icons in response to direct manual manipulation by a 
user; and 

icon rotation means connected to the pointing means and to the 
5 cursor means and responsive to a position of the cursor inside the display 

container for changing the shape of the cursor at the position to the shape 
of an icon of the plurality of icons and for changing the shape of the cursor 
to the shapes of other icons of the plurality of icons, such that the shape 
of the cursor is changed, one shape at a time in the predetermined order, 
10 in response to direct manual manipulation of the pointing means. 

9. The combination of Claim 8, further including means for 
illuminating icons displayed in the control panel in response to changing the shape 
of the cursor while the cursor is within the display container, wherein the means 
for illuminating illuminates an icon whose shape is the shape of the cursor. 
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